To address the problems of low accuracy， difficult deployment and high calibration cost of visual manipulator in complex system environments， a robust joint modelling and optimization method for visual manipulators was proposed. Firstly， the subsystem models of the visual manipulator were integrated together， and the sample data such as servo motor rotation angles and manipulator end-effector coordinates were collected randomly in the workspace of the manipulator. Then， an Adaptive Multiple-Elites-guided Composite Differential Evolution algorithm with shift mechanism and Layered Optimization mechanism （AMECoDEs-LO） was proposed. Simultaneous optimization of the joint system parameters was completed by using the method of parameter identification. Principal Component Analysis （PCA） was performed by AMECoDEs-LO on stage data in the population， and with the idea of parameter dimensionality reduction， an implicit guidance for convergence accuracy and speed was realized. Experimental results show that under the cooperation of AMECoDEs-LO and the joint system model， the visual manipulator does not require additional instruments during calibration， achieving fast deployment and a 60% improvement in average accuracy compared to the conventional method. In the cases of broken manipulator linkages， reduced servo motor accuracy and increased camera positioning noise， the system still maintains high accuracy， which verifies the robustness of the proposed method.